Systems, devices, and methods for augmented reality

ABSTRACT

Methods, devices, and systems are disclosed for providing augmented realities including trails or paths for navigating a real world space. Methods, devices, and systems are also disclosed for providing augmented realities for other forms of navigation guidance or tracking assistance.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/658,904, filed Apr. 17, 2018, the complete contentsof which are herein incorporated by reference.

FIELD OF THE INVENTION

The invention generally relates to augmented reality and, in particular,to forms of navigation guidance and tracking assistance using augmentedreality representations.

BACKGROUND

Navigational tools have advanced considerably from days when paper mapswere the only option for mobile persons to plan trips or draw routestaking into account a series of different locations. Portable electronicdevices have grown into a mass consumer technology complete with “GPS”(Global Positioning System) technology. Single-purpose instruments forselecting road directions proliferated under brands like Garmin®.Multipurpose instruments like smartphones have since proliferated toprovide consumers digital maps with driving direction features. Thoughmultipurpose devices like smartphones may perform other tasks besidesgiving driving directions, the navigation applications of smartphonesoffer essentially the same features and user experience assingle-purpose GPS navigation devices they've come to displace. Littlehas changed and improvements can feel marginal.

Between 2010 and 2020, virtual reality (VR) and augmented reality (AR)devices and technology rose substantially in use and availability amongordinary consumers and some businesses. However, in many instances suchdevices are limited to home use in contexts such as video gameentertainment. VR and AR remain underdeveloped as meaningful navigationguidance and tracking assistances tools.

SUMMARY

A target such as a human or vehicle may travel through the real world,and a user of an exemplary system or device is provided an ability totrack that target and receive visual AR content which portrays a paththe target leaves behind.

At a high level, the AR portrayal of a target's movement may take theappearance of a trail of “breadcrumbs,” an analogy to the familiarnursery story of children leaving a trail of breadcrumbs as a means formemorializing a path with some perpetuity. AR content of abreadcrumbs-type may entail the creation of specific augmentations whichtrace path or trails along which one or more targets previouslytraveled.

At a high level, the AR portrayal of a target's movement may take theappearance of a trail of fallen “dominos,” an analogy to the familiartile based game in which a path of dominos fall over among an initialplaying table full of standing dominos. AR content of a dominos-type mayentail showing many augmentations at the outset and removingaugmentations to signify a target has visited location corresponding tothose augmentations.

In both categories of embodiments—breadcrumbs-type anddominos-type—augmentations may be changed (e.g., altered in appearance)instead of being outright created from nonexistence or outright removedfrom existence.

A target's particular parameter that is tracked may differ amongembodiments. For example, one or more of location, movement, speed,latitude, longitude, altitude, depth, and proximity may be tracked,among other parameters. Changes in one or more of such parameters may betracked.

Proximity of a target to a particular real world location may be trackedsuch that the value of the proximity at multiple sample times isrecorded in, by, or with a virtual object associated with the particularreal world location. The proximity values thus stored may be then beretrieved at some substantially concomitant time or, alternatively, somefuture time for the purpose of changing augmentations of an AR outputbased on the changes in the stored proximity values. Proximity is oftenassessed between a mobile target an a stationary mobile referencelocation. However, other types of references (besides a referencelocation) may be used in some embodiments.

According to an aspect of some exemplary embodiments, methods, devices,or systems generate augmented reality (AR) output involvingproximity-based creation, destruction, and/or modification of ARcontent. In some embodiments, the AR content is further affected by atarget's view (e.g., viewing direction or frustum) at locations or overtime. In some embodiments, the AR content is further affected by auser's view (e.g., viewing direction or frustum) at locations or overtime.

By virtue of being created, removed, or otherwise changed, augmentationsmay have a temporary nature. According to an aspect of some exemplaryembodiments, a specialized system for storage and retrieval ofinformation is provided which facilities the creation, removal, andmodification of either or both breadcrumbs-type and dominos-typeaugmentations. According to such an exemplary system, a virtual worldmay be provided which is modeled after the real world.

According to some exemplary embodiments, the storage of locations whichcollectively define a path or trail is performed using a 3D virtualmodel of a real world space. In this disclosure, the terms “virtualmodel” and “virtual world” may be used interchangeably. An exemplary 3Dvirtual model has virtual locations which are configured to correspondwith real world locations. In other words, the 3D virtual model includesa virtual landscape modeled after the real world landscape. Real worldgeography, locations, landscapes, landmarks, structures, and the like,natural or man-made, may be reproduced within the virtual world in likesizes, proportions, relative positions, and arrangements as in the realworld. For example, a 3D virtual model of New York City would in factresemble New York City in many respects, with matching general geographyand landmarks. Within the virtual world, virtual objects may be created(e.g., instantiated) at virtual locations. Since a virtual locationcorresponds with a real world location, a virtual object at a givenvirtual location becomes associated with a particular real worldlocation that corresponds with the given virtual location. Data storedby or with the virtual object is also inherently associated with theparticular real world location. In some cases a single virtual objectmay be added as means for storing information for more than onelocation.

A virtual object stored in, with, or with reference to a virtual modelmay not inherently take a particular state as far as sensory modalitiesare concerned. For example, a virtual object may not have a particularappearance. Indeed, a virtual object may have no appearance at all, andin essence be “invisible” to an unaided human eye. By contrast, anaugmentation is by definition perceptible according to one or moresensory modalities. That is, an augmentation may be seen, heard,touched, smelled, and/or tasted. An augmentation may be regarded as the“face” of a virtual object, in which case data stored in, by, or withthe virtual object is used to determine what the augmentation portraysor signifies to a user looking upon that “face”.

According to one aspect of some embodiments, AR content is included inan AR output at multiple locations which were at some point in time inclose proximity to a tracked real world object like a person or mobileelectronic device. Wherever the tracked object went over some timeperiod of interest, locations proximal to the tracked object at variouspoints in time may be marked in the AR output with some augmentation,e.g., a virtual sign post. Proximity information may be stored forportrayal in AR content at some future time, in which case the proximityinformation may be stored (e.g., with a timestamp) using a virtualobject and retrieved from the virtual object at the time of ARproduction. As the proximity of a given location and the tracked objectchanges, e.g. the tracked object moves away, the augmentation may bemodified in the AR output (and/or in the virtual object) based on thechanging proximity. Virtual objects may be used to keep a record ofproximity over time, e.g., with different proximity values each having adifferent timestamp.

As an illustrative example, augmentations such as virtual sign posts maybe associated with real world locations. Conceptually this relationshipbetween augmentations and real world locations may be analogized to milemarkers on a highway, boundary pegs or boundary posts used by propertysurveyors, or signs marking street addresses. All such real objectsdesignate a real world physical location at which they exist. Incontrast to these real world sign posts, however, virtual postspresented to a user in an AR output may convey not just an identify of alocation, but also signify that a tracked target was near or at thelocation identified by the sign post. In other worlds, while real worldobjects like mile markers are strictly location-based, virtual objectsand their augmentations according to some exemplary embodiments may beboth location-based and proximity-based. AR content may be added,removed, or otherwise modified at specific non-mobile real worldlocations in dependence on the proximity of a mobile real world object(a target) with respect to those non-mobile real world locations.

For example, a sign post augmentation may be displayed in AR output forevery location a mobile device visits. As time elapses since the mobiledevice's last visit, the sign post augmentation may change appearance(e.g., fade or change color) to signify the passage of time since themobile device's last visit. As another example, as the distance growsbetween the mobile device and the location of an augmentation, theaugmentation may change (e.g., change size or shape) in dependence onthe changing proximity distance.

The preceding sign post examples are breadcrumbs-type AR. By contrast,according to the dominos-type AR, AR content may be removed from an ARoutput at each location which is or has been in close proximity to atracked real world object since some selected start date/time. In otherwords, some embodiments may involve producing AR content in which thepresence of augmentations (or the presence of augmentations of firstparticular appearance and not some other second particular appearance)signifies that the location has not been visited by a tracked target.Loosely analogous is the classic arcade game, Pac-Man, by which virtualyellow dots are removed from locations which Pac-Man visits. In Pac-Man,the presence of a yellow dot signifies that the location has not beenvisited by Pac-Man. The absence of an augmentation in the AR content (orthe use of alternative appearances to the augmentation) may signify thatthe location has in fact been visited by the tracked target within somepreceding window of time. In Pac-Man, the absence of a yellow dotsignifies that the location was already visited by Pac-Man. Othervariations may exist in other embodiments.

According to an aspect of some exemplary embodiments, augmentedrealities are provided in which a user is supplied AR content containingvirtual “breadcrumbs” which mark a path (i.e., trail) to follow in areal world view.

According to another aspect of some exemplary embodiments, augmentedrealities are provided in which a user is supplied AR content containingvirtual “dominos” which differentiate real world physical locationswhich have been visited from real world physical locations which havenot been visited.

According to another aspect of some exemplary embodiments, virtualtrails are generated using virtual augmentations to a real world view,where the virtual trails are created in response to a tracked target(e.g., a mobile electronic device) moving through a physical landscape.As the device moves from a first location to a second location to athird location and so forth, virtual objects are added to or updatedwithin a 3D virtual model of that real world landscape at virtual worldlocations matching the real world locations. In effect, the mobileelectronic device drops “breadcrumbs” in the form of virtual objectsalong the route navigated by the mobile electronic device. The virtualobjects are thereafter usable to generate augmentations which allow auser to visually retrace the path of the tracked target that left the“breadcrumbs”.

Exemplary embodiments of the invention may involve a wide array ofapplications. AR trails of a breadcrumbs-type provided by exemplarymethods and systems may provide guidance to hikers, bikers, skiers, andother outdoorsmen when they are lost or disoriented. AR trails may beprovided for use by law enforcement personnel (e.g., police officers)when, for example, chasing a suspect or investigating (e.g., recreating)past events. Responding officers arriving at the initial scene of acrime may be provided with AR trails following an officer already inpursuit of a suspect. AR trails may be provided for replaying trainingscenarios. AR trails may be provided to mark a path to a particulartarget, destination, or other user.

AR trails of a dominos-type also have a variety of possibleapplications. For example, in the area of public safety, a dominos-typemethod and its AR output may easily highlight areas that have or havenot been searched. As another example, in the area of wireless networksor surveying, the method and its AR output easily highlight areas thathave or have not been measured. As yet another example, in a militarycontext, the method and its AR output may assist in the clearing ofminefields. Locations at which an initial AR augmentation has beenremoved and therefore which is no longer visible have been cleared,whereas locations at which AR augmentations are still visible remain tobe unsafe.

Advantages of exemplary embodiments are multifold. Exemplary AR contentmay provide a relatively passive form of guidance that is neitherintrusive or excessively distracting of a user's attention. For example,a simple path formed by visual cues or markers dotting a path of“breadcrumbs” provides a user simple and intuitive visual guidancewithout excessive distraction. Furthermore, in some embodiments, ARtrails may easily display many markers within a user's viewing frustumwithout a risk of overwhelming, inundating, or confusing the user. Auser may be provided with a readily understood visual basis forassessing not only the most immediate movement required but alsomovements in the future which may be in visual range.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of an embodiment for producingproximity-based AR content.

FIGS. 2A to 2F are schematic illustrations of an exemplary embodimentfor producing proximity-based AR content of a breadcrumbs-type.

FIG. 3A to 3Q are schematic illustrations of an exemplary embodiment forproducing proximity-based AR content of a dominos-type.

FIG. 4 is an exemplary method.

FIG. 5 is another exemplary method.

FIGS. 6A to 6F are illustrations of an exemplary embodiment.

FIG. 7 is a block diagram of an exemplary system.

FIGS. 8A and 8B are opposite sides of an exemplary AR-enabled device.

FIG. 9 is a frame of an augmented reality output in which an urban areahas been augmented with path marker augmentation.

FIG. 10 is a frame of an augmented reality output in which a wooded areahas been augmented with trail marker augmentations.

FIG. 11 is a frame of an augmented reality output of a park at nightwhich has been augmented with trail marker augmentations.

FIG. 12 is a frame of an augmented reality output of a dividing biketrail which has been augmented with trail marker augmentations.

DETAILED DESCRIPTION

FIG. 1 schematically illustrates an exemplary embodiment that producesproximity-based AR content. In this example, a real world person 101 hasa mobile electronic device 102 (e.g., a mobile phone, wearable, laptop,smartwatch, etc.). The person 101, the device 102, or some combinationof the pair constitutes a real world target, and an exemplary method orsystem involves tracking the real world target as the target changeslocation in the real world. For simplicity and clarity of illustrationin FIG. 1, locations and movement are confined to a unidimensionalphysical space (specifically a line, with possible directions ofmovement being left or right on the page). The person 101 and device 102are mobile and move to different locations as time progresses. Time isrepresented in increments, with four instances of time illustrated,labeled respectively as T₁, T₂, T₃, and T₄. Two real world locations arealso illustrated and labeled. They are identified as Loc A and Loc B,respectively. An exemplary system may comprise the mobile device 102 andprocessors (not shown in FIG. 1), among other support hardware, thecombination of which is configured to track the proximity of the target(user 101 and/or device 102) with respect to one or more real worldlocations. The system is further configured to modify the AR contentoutput from the system based on the target's proximity to real worldlocations. In particular, the system may change augmentations of an ARoutput based on one or more changes in the proximity of the trackedtarget to the real world locations, Loc A and Loc B. To affect thetracking and changing, the real world location under consideration (LocA and Loc B) may each be associated with virtual objects which in turnmay sometimes (but not at other times) be associated with respectiveaugmentations.

For convenience of discussion, that with which the target is comparedmay be referred to as a reference. The target may be a person, anobject, a location, or some other thing (typically but not necessarilytangible, and typically but not necessarily having recognizableboundaries). The reference may be a person, an object, a location, orsome other thing (typically but not necessarily tangible, and typicallybut not necessarily having recognizable boundaries). A target may be avehicle, a device such as a mobile electronic device (e.g., a mobilephone, wearable, laptop, smartwatch, etc.), an animal, a person of aparticular type (e.g., a criminal suspect, a law enforcement officer, asoldier, a civilian, a child, etc.), a user, some other thing which maymove from time to time, a plurality of any of these, and or acombination of any of these.

Proximity may be defined, characterized, or otherwise assessed in one ormore of a variety of forms. At a high level, proximity entails how closetogether or how far apart two items are. Proximity may be a constant inthe event a target and reference both maintain fixed positions within acommon frame of reference. Proximity changes when either the target orthe reference moves with respect to the other. For convenience ofdiscussion, examples herein tend to describe proximity changes on theassumption that the reference is fixed (location is constant) and thetarget is mobile and has changed location at least once in some timewindow of interest. This is a non-limiting scenario used for exemplaryillustration.

In FIG. 1, locations are defined according to a proximity range (e.g.,A-A or B-B). In this example, proximity is characterized as a binary:either the target is “at” the reference, or the target is “away from”the reference. Locations Loc A and Loc B are references, device 102 is atarget, and ranges A-A and B-B are elements which facilitatecharacterizing the proximities among the references and target. Thedevice 102 may be at Loc A by falling within the distance range A-A, andthe device 102 may be at Loc B by falling within the distance range B-B.Conversely, if the device is outside a range A-A or B-B, the target isaway from the respective reference.

References (like reference locations Loc A and Loc B) are associatedwith virtual objects. Said differently, virtual objects are associatedwith the references. In FIG. 1, virtual object 111 is associated withreal world location Loc A, and virtual object 112 is associated withreal world location Loc B. An exemplary method of creating suchassociations is to instantiate or set such virtual objects as 111 and112 within a virtual model (virtual world) which is configured to modelthe real world. Such virtual models create virtual spaces which resemblethe real world. In particular, real world geography, locations,landscapes, landmarks, structures, and the like, natural or man-made,may be reproduced within the virtual world in like sizes, proportions,relative positions, and arrangements as in the real world. Within thevirtual world, any number of virtual objects may be created (e.g.,instantiated) at virtual locations. Since a virtual location correspondswith a real world location, a virtual object at a given virtual locationbecomes associated with a particular real world location thatcorresponds with the given virtual location. Data stored by or with thevirtual object is also inherently associated with the particular realworld location.

Storing information in virtual objects offers certain advantages. Oneadvantage is the option of permanency. Central to many embodiments ofthe invention are changes which are made to augmentations to reflectchanges in real world conditions, e.g., the movement of a real worldtarget. As a result, augmentations may be temporary, even fleeting.Augmentations may come and go, and change so dramatically in appearanceor by other sensory modality that information formerly portrayed by theaugmentation is all but lost in an updated state of the augmentation. Bycontrast, virtual objects may persist where augmentations do not. Ineffect, virtual objects may provide a constant backbone for a variety ofdifferent and changing augmentations. Virtual objects may serve as datastores comprising a compilation of information for a respective realworld location. An augmentation associated with a particular virtualobject may be based upon just a fraction of the data maintained by thevirtual object. That data which is not drawn upon may be preservedvirtually, permitting it to remain available for a future change to theaugmentation.

Virtual objects 111 and 112 have locations within the virtual modelwhich match the real world locations with which the virtual objects areassociated (here, Loc A and Loc B, respectively).

By definition virtual objects are entities of virtual reality and/oraugmented reality which may or may not be “visible” or otherwiseperceptible to a human (e.g., audible, tactile, etc.). At the time ofwriting this disclosure, the most common augmentations in AR content arevisual and/or audial, and many of the illustrative examples willdescribe visual and/or audial augmentations. It should be understood,however, that additional or other sensory modalities may be employed inthe practice of the invention (augmentations may be one or more ofvisual, audial, tactile, gustatory, and olfactoral). An “augmentation”is a sensory output of an AR or VR system by which a virtual object isexpressed to a human. In the example of FIG. 1, virtual objects 111 and112 are illustrated with broken lines for the benefit of a reader ofthis disclosure, but the person 101 does not perceive the existence ofanything virtual at Loc A or Loc B at T1 because no augmentations existat T1.

If the target (here, device 102 or, indirectly, person 101) comes withinthe proximity range of a location (Loc A or Loc B), an augmentation iscreated or modified as a result of the change in proximity. At time T1,however, device 102 is not at Loc A or Loc B. Accordingly, an AR outputmay not show any augmentation at Loc A or Loc B. Of course,augmentations which are unrelated to this method of tracking proximityof a target may still be displayed.

Between T1 and T2 person 101 and device 102 moved. As a result, at timeT2, device 102 falls within range A-A and is therefore at Loc A. Asystem detects this new proximity state (the proximity state at T2) and,as a result, the AR output is modified. An augmentation 121 (here, whatappears as a darkly shaded rectangular post) now appears in the ARoutput at Loc A. The change in the tracked target's proximity to Loc Abetween T1 and T2 results in the change of AR content for Loc A. Whereasno augmentation existed at T1, an augmentation 121 now exists at T2. Theaugmentation 121 is a perceptible output corresponding with the virtualand otherwise imperceptible virtual object 111.

Between T2 and T3 person 101 and device 102 move again. At time T3, thetargets are neither at Loc A or at Loc B. The change in proximity to LocA (e.g., the change from the user being proximal to Loc A at T2 to theuser no longer being proximal to Loc A at T3) has been detected by thesystem. In response to this detected change, the system changes the ARoutput to modify the appearance of the augmentation 121 displayed at LocA. Now the AR output shows augmentation 131 at Loc A. Augmentation 131is a modification of augmentation 121 and, as apparent from FIG. 1, isshaded with a medium density shade instead of a high density shade. Thischange in the AR output, from augmentation 121 to augmentation 131, isan important feature of this exemplary embodiment. The change in ARoutput provides an indication (here, a visual indication) to a user thata change in proximity has occurred. In particular, the change from adark shade augmentation 121 to a medium shade augmentation 131 isindicative that the target (device 102 or user 101) was at Loc A at somepreviously point in time but has since moved away from Loc A.

Between T3 and T4 person 101 and device 102 move again. As shown in FIG.1, at time T4, the target (device 102 or user 101) is now within therange B-B and therefore determined to be at Loc B. This proximity changefor Loc B is detected and triggers a modification to the AR content forLoc B. In particular, the system now displays an augmentation 142 at LocB which indicates to a user that the proximity state of the target (user101 or device 102) has changed with respect to a reference (Loc B). Inthis example, augmentation 142 matches augmentation 121 in appearanceattributes (that is, both are darkly shaded posts) because thisparticular visual VR output is used in this example to indicate to auser that the target is at the location where the darkly shaded postappears. Augmentation 142 is associated with virtual object 112.

The preceding paragraph focused on the proximity change of the targetwith respect to reference, Loc B. Between T3 and T4 other proximitystates are changing as well, with respect to other references. In thislimited example, the other reference is Loc A. From FIG. 1 it isapparent that the VR output for Loc A has changed from a medium shadeaugmentation 131 to an augmentation 141 which has light shading or noshading at all. While it was stated above that this example involvesonly two possible proximity states—“at” or “away”—these states may havesub-states which provide additional information through the VR output.As an example, the augmentations may be used to indicate to a user theamount of time a target has been at or away from a reference. In anon-binary system (e.g., a system with three or more qualitative orquantitative states; a system in which proximity is defined metricallyaccording to a measured distance between target and reference), thechanges in augmentations may be used to indicate how far away or howclose a target is to a reference.

In FIG. 1, the change in appearance of the augmentations at Loc A fromT2, to T3, to T4 is a predetermined setting used to signify to a viewerof the AR content an amount of time that has elapsed since the target(user 101 or device 102) was at Loc A. As between the three times (T2,T3, and T4), a viewer of the AR content would be able to understand thatat T2 the target was at Loc A, that at T3 the target was at Loc A arelatively short time ago (but no longer), and at T4 that the target wasat Loc A a comparatively long time ago (but no longer). After apredetermined duration of time has elapsed in which the target hasfailed to return to Loc A, the augmentation at Loc A may be removedentirely (essentially resuming the same omission of any appearance as atT1). If the user were then to return to Loc A, the augmentation wouldonce again be changed and a timer associated with the augmentationreset.

In FIG. 1, every reference location (Loc A and Loc B) is permanentlyassociated with a virtual object (111 or 112). While the augmentationassociated with the virtual objects were generated, changed, or removed,the virtual objects themselves continued to exist. This is but onepossible approach to implementing a method of tracking proximity of atarget. Other alternatives are possible. For instance, virtual objectsmay be created and removed concurrently with the creation or removal oftheir corresponding augmentations.

In FIG. 1, every reference location also has a one-to-one relationshipwith an augmentation. At any time (T1, T2, T3, or T4) each augmentationwas associated with at most one virtual object. This is but one exampleconfiguration and others are of course possible. For instance, assuminga series of point locations serve as references, and each reference isassociated with a virtual object, the collection of virtual objects maytogether correspond with one augmentation such as a continuous visualband or path that does not contain visual “breaks” from one location orvirtual object to the next location or virtual object.

FIGS. 2A to 2F are illustrative of another exemplary method bearing manyof the same qualities as the method illustrated by FIG. 1. Notablydifferent, however, is that FIGS. 2A to 2F illustrate space intwo-dimensions, and the system is configured such that thetwo-dimensional real world plane is divided into a plurality oflocations of predetermined width and length. All the locations togetherconstitute 100% of possible real world space in the two-dimensionalplane. Whereas in FIG. 1 it was possible for the target to be at noreference location (e.g., the scenarios at times T1 and T3), in FIGS. 2Ato 2F the target is at all times at one and only one reference location.All possible physical space has been attributed to one referencelocation or another. As illustrated, the system is configured torecognize sixteen reference locations (A1, A2, A3, A4, B1, B2, B3, B4,C1, C2, C3, C4, D1, D2, D3, and D4). Each figure from FIG. 2A to FIG. 2Frepresents an incremental change in time. Thus, FIG. 2A captures a starttime, FIG. 2F captures an end time, and FIGS. 2B to 2C show changes inAR output occurring between the start time and end time.

In FIGS. 2A to 2F, an augmentation is created for each location thetracked target visits. Thus, by simple observation of which locationshave a virtual post and which do not, the AR output identifies for auser which locations the target has visited and, conversely, whichlocations the target has not visited (during a particular window oftime). Each augmentation (here, posts) functions as a virtual“breadcrumb” left behind by the target (here, person 201 or device 202),thereby tracking his path through the real world space (here, the realworld space is characterized by the illustrated grid of sixteen blocks).

The path (or paths) involved in method 500 may be any real world path,such as along or through bridges, roads, and buildings, or such asacross country through a field or woods. A tracked target (such as aflying vehicle like an airplane, or a person using on such a vehicle)may be capable of flight, and a tracked target (such as with asubmersible vehicle like a submarine) may be capable of diving. In theseamong other cases paths may especially involve altitude or depthconsiderations. In the alternative to receiving locations from a knownorigin, the system may receive a plurality of locations the origin ofwhich is not necessarily known and/or consequential. In any case, theplurality of locations may describe a path used or useable to physicallytraverse a real world landscape. For instance, the locations may each bea fixed number of meters apart (1 meter, 10 meters, 20 meters, etc.)from an adjacent location in the same set and collectively trace out apath from a starting location to a destination location.

A target's location or proximity to one or more references may betracked at different resolutions depending on the embodiment. In FIGS.2A to 2F, the tracking resolution is based on location resolution.Locations may also be treated as points, which may furthermore haveintervening space. This treatment of location is consistent with theexample illustrated by FIG. 1. Locations may also be treated as areas.This treatment of location is consistent with the example illustratingby FIGS. 2A to 2F. In the case that 100% of real world space is dividedamong discrete locations, the larger the number of locations, the higherthe location resolution. Conversely, the fewer the number of locations,the lower the location resolution. Using numbers to illustrate, if eachlocation of a 2-D grid based system like that of FIGS. 2A to 2F measures10 m×10 m, the tracking resolution would be 100 times less than it wouldif each location of the 2-D grid based system measured 1 m×1 m. Thisaspect of resolution of tracking is inherent to existing systems oflocation determination, e.g. global positioning systems (GPS).

The resolution of an AR trail formed by one or more augmentations mayalso vary, in part due to the resolution of the definition of locations,and in part due to other factors. As a very simple example, in FIGS. 2Ato 2F, as many discrete augmentations were produced as locations werevisited by the tracked target. However, it may that in a variantembodiment, only every other location visited would be represented withan augmentation. It may be that a single augmentation is produced torepresent a grouping of multiple locations. It may be that augmentationsare produced on the basis of time, e.g., the location of the trackedtarget is represented with an augmentation at every increment of time(say every 60 seconds) and any changes in location in the intervening 59seconds are either not tracked, not represented as augmentations despitebeing tracked, or both. As was the case in the example of FIG. 1, inFIGS. 2A to 2F, the visual appearance of augmentations involves a shadewhich changes from darker to lighter as the proximity of the user to agiven location diminishes and/or as the time since the user was at alocation grows. This is but one example of how augmentations may changein dependence on proximity.

FIGS. 3A to 3Q illustrate another exemplary method for producing ARcontent that is proximity-based. In the examples of FIGS. 1 or FIGS.2A-2F, the presence of an augmentation associated with a real worldlocation signified to a user that a tracked target had visited that realworld location within some time period of interest (along with possibleother information about the tracked target's travel history). Theexample of FIGS. 3A to 3Q shows the inverse, whereby the presence of anaugmentation associated with a real world location may signify that auser has not visited a location within some predetermined time period ofinterest. In this case, AR content may contain one or more augmentationsthe presence and/or appearance of which signifies that the location (orlocations) with which the augmentations are associated have not beenvisited within some predetermined period of time. In FIG. 3A, of 16possible locations (A1, A2, A3, A4, B1, etc.), each is associated in theAR output with an augmentation (in this case, virtual posts). Each stepfrom one figure to the next figure in the series of 3A to 3Q representsa step in time. As can be appreciated by viewing each of the figures inturn (especially as one would view a flipbook) the change in proximityof a target (either person 301 or device 302) with respect to alocation, from being away from a location to being at a location, causesa change in the augmentation associated with the location in question.In this example, the change comprises the removal or deletion of theaugmentation altogether, although other changes (e.g., a modification ofappearance such as from one color to another color) may occur instead ofoutright removal of an augmentation.

Methods like that illustrated by FIGS. 3A to 3Q are particularlyadvantageous when, for example, the method is for clearing an area. As aprime example, law enforcement and security personnel may requirechecking a plurality of locations on a circuit or rotation to ensurepublic safety or private security of real property. An AR output likethat of FIGS. 3A to 3Q gives an intuitive visual basis for assessingwhich locations have been cleared and which have not, based on whetheror not an augmentation has been cleared away, or whether theaugmentation remains unchanged from a default state signifying alocation as unvisited within some time period of interest.

Thus far both breadcrumbs-type methods and dominoes-type methods havebeen described in such a way that proximity of a target to a referenceis the main determinant, perhaps the only determinant, of whether anaugmentation is added, removed, or otherwise changed. This need not bethe case in all embodiments. Another condition or trigger which may leadto a change in AR content is a target's view. In particular, it may notonly matter where a target physically travels, but where the target'seyesight “travels”. In short, augmentations of an AR output may bechanged based on one or more changes in proximity of a tracked target toa reference and/or the target's view of a reference.

The significance of the view criteria for determining whether to changean augmentation is illustrated well by the scenario of law enforcementor security personnel clearing locations while on patrol. A patrol maybe conducted with a patrol car, from which an officer inside takesvisual stock of the car's surroundings but does not physically leave theroadways or drive down every roadway (e.g., say a short alley connectedto a main roadway). In this case, the physical presence of an officer inthe alley may not be of particular criticality provided the officer'sline of sight is able to reach the alley from one end. Thus, in an ARoutput, if the patrolling officer drives past the opening of an alleywaywithout directing his view into the alley, an augmentation correspondingto the alley may not be changed. By contrast, if the patrolling officermakes the same drive but directs his view into the alley, anaugmentation corresponding to the alley may be changed or removed.

Another criterion which may be assessed to determine changes toaugmentations is whether a particular action is taken by a target whenat a reference location. For example, the consumer electronics show(CES) is a highly anticipated annual tradeshow (at least as of thewriting of this disclosure). The CES, like many tradeshows, may involvebooths at which respective companies showcase their products. A reporterat such an event must take particular actions in connection with heroccupation—capture photographs or video content, conduct interviews, orread informational postings, for example. The proximity of a reporter(the target in this example) to a particular booth (a reference) may beof some interest to assess the reporter's coverage, but the actionstaken by the reporter when at a booth are of particular consequence intheir own right. For such contexts where not just locational proximityis of interest but also actions taken at locations, embodiments may beconfigured so that AR content is updated to signify what actions havebeen performed, or what actions have not been performed from somepredetermined list. An action which may be signified by an augmentationmay be any of a variety of actions. For example, the action may betaking a picture, sending an email, transmitting a signal (e.g., makinga phone call), receiving a signal (e.g., receiving a phone call),activating a device, deactivating a device, etc.

The visual appearance of an augmentation may be configured to signify toa user an aspect of time. For example, the appearance may signify howold the augmentation is since it was created or since it was changed.The appearance may signify a date and/or time at which the augmentationwas created or previously changed. For example, visual appearance of oneor more augmentations may change opacity (e.g., fade) or change size(e.g., shrink) as time elapses, or as the augmentations age. A color,shade, or opacity of a visual augmentation may be configured to signifythe amount of time since a target was at each augmentation's associatedlocation. The visual appearance of the augmentations may indicate, forexample, when the augmentation was created, how old the augmentation is,and/or how close or far the target is from the augmentation in realtime.

The visual appearance of an augmentation may signify an aspect of speed.For example, if a tracked target passes a real location for which anaugmentation is provided, the augmentation may visually signify thespeed with which the tracked target passed the location. The visualappearance may also or alternatively give directional information. Forexample, the augmentation may include or signify a vector (e.g., arrow)which signifies the direction the target was moving when passing thelocation.

The visual appearance of an augmentation may signify the proximity ofthe associated real world location with some other real world location.For example, individual augmentations may indicate the measure ofdistance between the associated real world location and an end location,such as the end of a trail to which the augmentation contributes avisual path of “breadcrumbs”.

In breadcrumb-type embodiments, multiple trails may be displayedsimultaneously in AR content. In such cases, the augmentations ofrespective trails may be provided with different visual appearances todifferentiate one trail from another. In dominos-type embodiments,multiple trails may be displayed simultaneously in AR content. Thedifferent targets which give different trails may or may not bedifferentiated in the AR output. For example, if law enforcementofficers are clearing an area, identifying the particular officer whoclears an area may not be important, in which case the AR content may beconfigured to show only that locations are cleared (by any officer)without encumbering the AR content with information conveying whocleared each location.

The visual appearance of an augmentation may be configured to signifydwell time. Specifically, an aspect of the augmentation's appearance maycorrelate with the amount of time a tracked target spent at the locationcorresponding with that augmentation. As one example, augmentationsshowing a trail of a tracked target may be portrayed as drops of liquidleft behind by the target, not unlike the manner in which a personsoaked by rain leaves a trail of drops in their wake as they passthrough a hallway. If the target remains at a location for a protractedperiod, the augmentation may comprise a pool of droplets, and a size ofthe puddle may qualitatively show the duration of time the targetdwelled at the associated location. Other visual representations ofdwell time, either qualitative or quantitative, may be used depending onthe embodiment.

In some cases, the identity of the target may be unknown or deliberatelyhidden from a user. A trail left by the target may nonetheless bepresented. In some embodiments augmentations may be configured tosignify to a user an identity of one or more targets and/or other users.A trail of augmentations may, for instance, comprise information whichidentifies a person or object which left the trail. As an example, thisfeature may be used by personnel who must verify, corroborate, orotherwise check the work of other personnel. In the dominos case, if aninferior is required to visited particular locations as part of apatrol, a change in augmentations to indicate the inferior visited(versus no one visiting or someone who is not the inferior visiting)provides a straightforward means for the superior to ascertain theperformance and completeness of the inferior's patrol.

The AR content may be determined or changed based on location orproximities of a second tracked object (besides the target). Inparticular, a user's location may be tracked and the AR content suppliedto the user changed based on the user's location. For example, oneaugmentation “crumb” may be made visible at a time. When a user reachesthat augmentation (i.e., reaches the location associated with theaugmentation), the next augmentation in the trail becomes visible. Asanother example, the next augmentation in a sequence may have a uniqueidentifier that all others do not. For instance, the next augmentationmay be blinking while the remaining augmentations are still. As theusers's location changes as the trail is followed, which augmentation isblinking is updated and changed.

In most embodiments a target (or targets, as the case may be) movethrough a real three-dimensional real world space. The real worldlocations a target visits affect virtual objects and correspondingaugmentations which are associated with the real world locations throughwhich or past which the target actually moves. The same is notnecessarily true of users. While users may also move through a realthree-dimensional real world space, users may also take advantage of thesystems and methods herein from very different locations or vantages.For example, a user may be at a remote location from a target yet besupplied VR or AR content which shows a breadcrumbs-type or dominos-typetrail for the target. A user may be entirely stationary, e.g.,positioned at a desktop at a command center, in a squat car, in a datacenter, etc. Of course, the ability to provide AR content to a user inthe same real world setting through which a target has previously passedis particularly advantageous. However not all embodiments necessarilyconform to this modality of content output.

Different users may be provided different augmentations, even in caseswhere the augmentations are based on the same virtual objects. Forexample, different information or different kinds of information may beshared with different users. Users may be assigned or attributeddifferent clearance levels (e.g., security levels), and augmentationsselectively chosen for output to a particular user based on that user'sclearance level.

Significant time may or may not elapse between i) a time when a targetvisits, passes through, or passes by a location and thereby triggers achange in a virtual object and/or augmentation for that location, andii) a time when a user consumes the VR or AR content generated at leastpartially based on the target's visit or passing. The date/times (i) and(ii) may be both substantially in real time. The date/times (i) and (ii)may be substantially delayed, e.g., minutes, hours, a day or more, aweek or more, or a month or more. Indeed, some embodiments may beemployed for visually recreating events of the past. In a criminalinvestigation, for example, investigators may be supplied VR or ARcontent which permits them to visually observe the movement or pathtaken by a suspect at some time in the past that is of consequentialityto the investigation. As previously discussed, though individualaugmentations may be relatively fleeting, the use of virtual objects asa backbone for producing augmentations provides a basis for moreperpetual storage without loss of trail information tied to particularreal world locations. This modality of storing and retrievinginformation is both efficient and robust.

Several of the features described above will now be illustrated with anexample involving law enforcement officers (LEOs) pursuing a criminalsuspect. Assume a criminal flees a crime scene on foot. The criminal isthe tracked target. His movement is tracked, at least to some extent, byone or more of a helicopter keeping track of him from the air, a mobiledevice carried by the criminal the position of which has been estimatedby cell tower triangulation or a GPS signal, and street camerasrecording the criminal briefly as the criminal passes the respectivecameras. As the fleeing criminal takes some path through streets,alleys, buildings, etc., he (briefly) visits real world locations suchas a street corner, an intersection, a postal office, a particular pointidentifiable with GPS coordinates, etc. Visiting a location does notnecessarily require any more than a moment at the location. One or moreof these locations is associated with a virtual object, and theinformation for such virtual object is updated to indicate informationabout the criminal's visit, e.g., whether he visited or not, when hevisited, whether he visited for more than a moment, how long he dwelledif he dwelled, how fast he was going when he visited, which direction hewas traveling when he visited, etc. A law enforcement officer (LEO) onthe ground is in pursuit of the criminal and is following his trail. ARcontent is supplied to the LEO to assist his ability to understand whatpath the criminal took. Augmentations at some of the aforementionedlocations are created, with possible visual appearance characteristicsto signify some of the information previously listed, to inform the LEO.The real time location of the officers may be tracked so that theirproximity to the virtual objects is known. As a result, the LEO may beprovided AR content which only contains augmentations within a certainproximity of the LEO. As a result, the LEO is not inundated with the ARcontent for the criminal's whole path, but receives only thoseaugmentations to which he is nearest. Which augmentations are providedmay be updated as the LEO's location and proximities changes, keepingthe AR content relevant to the LEO in real time. Meanwhile a commandcenter may be provided with another form of AR content. At the commandcenter personnel may be working from stationary computers. The computersmay display real world content, such as footage from the helicopter orfootage from the street cameras, with virtual augmentations showing thebreadcrumb-type path created by the fleeing criminal. To the commandcenter, both the criminal and the LEO are possible targets of interest.Thus, the virtual content served to the command center for display mayinclude augmentations which also trace a path the LEO has taken and/oris in the act of taking. In this way the command center is provided theability to monitor both the fleeing suspect and the LEO who is on footchasing the suspect. The information concerning the suspect's path andthe LEO's path may both be stored using virtual objects in a virtualmodel. Sometime later, say one month, a criminal prosecutor orinvestigator may access the stored information. AR content is providedto the prosecutor or investigator the content provided to the commandcenter the night of the crime. The temporal gap between the events whichformed the trails and the serving of AR content is made possible by thesupporting virtual model and storage/retrieval system.

“User,” as used herein, is an entity which employs a method, device, orsystem of the invention. A user may be a human, multiple humans, or someother entity. A user may be, for example, a person intended to consumeAR content generated in accordance with a method or variant of a methoddisclosed herein. A user may be a person in pursuit of a mobile target.A user may be the target, such as would be the case that when a personwishes to retrace his or her steps and would benefit from AR showingwhere he or she previously visited, or what he or she did at visitedlocations, or how long he or she dwelled at visited locations, etc.

FIG. 4 is a flowchart for an augmented reality (AR) method 400 which maybe used to achieve AR outputs such those of the preceding exemplaryembodiments. Virtual objects may act as a backbone to the method in thatthey exist virtually yet are associated with real world locations (block401). This association permits augmentations to be accurately andrealistically placed in the context of the real world (i.e., in semanticcontext with the real world) when the augmentations are made based oninformation stored with the virtual objects. A trail, be itbreadcrumbs-type or dominos-type, may be based on the movement of sometarget, and therefore some aspect (or aspects) of the target aretracked. In exemplary embodiments, proximity of the target with respectto certain locations may be tracked (block 402). The proximityinformation may be stored with the virtual objects associated with thecertain locations. Based on changes in proximity recognized by thetracking step, augmentations of an AR output may then be changed toprovide a user a simple and intuitive basis for understanding thetarget's movement history, e.g., where the target visited, how long thetarget visited, and how long ago the target visited, among other typesof information as discussed throughout this disclosure.

FIG. 5 shows a flowchart for an augmented reality (AR) method 500.Method 500 includes steps for determining which virtual objects to usewhen determining what the AR content should include for a given user,based on that user's unique circumstances.

FIGS. 6A to 6F illustrate relationships between real world, virtualworld, and two alternative AR outputs for some exemplary embodiments.These figures will be referred to periodically to lend a visualunderstanding of the method of FIGS. 4 and 5. Individual figures aresnapshots of time, with six times being illustrated (T1, T2, T3, T4, T5,T6). The virtual world is modeled after the real world, thus theresemblance of the two worlds within the figures. While virtual worldsmay vary in size and scope, what is illustrated may be regarded asshowing just that part of a virtual world which corresponds with thatpart of the real world illustrated. For obvious constraints like papersize only a limited amount of the real world can be shown in thefigures. In various embodiments, virtual models may vary in in thedegree of detail which is replicated from the real world. For instance,a real world building may be reproduced merely as a box in the virtualworld, as a box with features like doors and windows, as a box withfeatures like doors and windows together with textures of respectivesurface, etc.

In FIGS. 6A to 6F, everything depicted in the row labeled “REAL WORLD”is real (e.g., tangible, comprising matter, reflecting real light wavesoff their surfaces, etc.). The exceptions to this are the traditionalpatent drawing labels, and the marking of some locations small “x”indicia for identifying locations to the reader of this disclosure.These “x”s do not appear as indicia in the real world.

In the row labeled “VIRTUAL WORLD”, everything is virtual. The virtualworld involves data characterizing the real world and which, whensuitably processed, permits an output such as what is shown in thefigures. Virtual objects 611, 621, 631, and 641 may or may not take avisual form in the virtual world. In FIGS. 6A to 6F, some virtualobjects are depicted as small cubes like object 611, 621, 631, and 641.The other items in the virtual world, including for example the trees612, building 613, and mountain 614, may also each exist by means of oneor more virtual objects.

The row labeled “AR1” shows augmented reality content for a first user.The row labeled “AR2” shows augmented reality content for a second user.Real world content is depicted in these rows using broken lines for easeof distinguishing real and virtual to a reader of this disclosure. Inactual implementations, real content and virtual content may beindistinguishable, as may be desired for highly realistic AR.

Returning to FIG. 5, method 500 is a method for providing an augmentedreality (AR) that may include AR paths (i.e., trails).

The exemplary process 500 describes how paths may originally be formedand stored as well as how paths may be retrieved for display in anoutput. At block 501, a plurality of real world locations are receivedthat collectively describe a path used or useable to physically traversea real world landscape. For example, this receiving step may comprisereceiving the plurality of real world locations from a mobile electronicdevice as a result of the mobile electronic device physically visitingthe respective real world locations. The mobile device may periodicallytransmit or store its location as it is moved by a user through a realworld geographic space.

At block 502, the locations from block 501 are stored for subsequentretrieval and use. If the locations of interest are already associatedwith virtual objects, information stored by the virtual object may beupdated.

In FIGS. 6A to 6F, the tracked target is a truck 610. In the real world,the truck 610 is driving along roadways from T1 to T4. With each step intime, a location is obtained for the truck describing the truck's path(consistent with block 501 of FIG. 5). Virtual objects 611, 621, 631,and 641 are added to the virtual world based on the received locations(consistent with block 502 of FIG. 5). The dotted lines 690 areindication to the reader of this disclosure the association of the realworld locations with the respective virtual objects in the virtualworld. Here new virtual objects are being added to the virtual world inFIGS. 6A to 6D and in block 502 of FIG. 5. According this approach, eachnewly created virtual object 511 has proximity of zero with respect tothe tracked target (here, the truck). Once the target moves away fromthat particular location and its corresponding virtual object, theproximity of the target with respect to the location and virtual objectincreases to a value greater than 0. Alternatives to this approachexist. For example, the four virtual objects 611, 621, 631, and 641 mayexist prior the start of a method 500 or the like. In other words, thefour virtual objects 611, 621, 631, and 641 may already exist at T1. Inthis case, block 502 may alternatively or additionally comprise updatingvirtual objects based on the tracked target's proximity with respect tothe location associated with each respective virtual object (or someother parameter besides proximity). Indeed, at any time after a virtualobject's creation, its proximity with respect to a tracked target may bemonitored and updated.

Blocks 503 to 505 of FIG. 5 involve retrieving a path from storage(e.g., memory) for subsequent use in an augmented reality output at auser's AR device (e.g., a head mounted display, a mobile phone, atablet, a laptop, a wearable device, etc.).

At block 503, a real world frustum is determined. This real worldfrustum is regarded as the user's viewing frustum, and may correspondwith the viewing frustum of a camera or cameras of an AR device whichcaptures real world image data describing the user's real worldsurroundings. A real world frustum may be determined based on one ormore of, for example, a present location (e.g., of the AR device), afield of view (e.g., of the AR device's camera), an orientation (e.g.,of the AR device's camera), a position (e.g., of the AR device orcamera), a pose (i.e., a combination of position and orientation), andassumptions about the near and far field limits (e.g., predeterminedvalues for near and far field limits).

At block 504, the determined real world frustum is applied to thevirtual world of the 3D virtual model. Essentially, the real worldfrustum is used to set the viewing frustum within the virtual world.Virtual objects which are inside the (now virtual) viewing frustum arefound as candidates for augmentation. Virtual objects lying entirelyoutside the viewing frustum are not candidates for augmentation.

At block 505, a selection of augmentations based on the virtual objectcandidates occurs. This selection may involve one or more criteriaincluding, for example, user option selections and the relationshipsbetween different virtual objects. In particular, virtual objectcandidates are selected which collectively describe a path or trail fromthe AR user's location (or approximate location) to some other locationof interest, generally referred to as a destination location.

In FIGS. 6E to 6F, a car 651 is both a tracked target and a user. Thecar 651 exists in the real world. For purposes of this discussion, thecar's real world location is marked with a black circle 652, and thecorresponding virtual world location with a black circle 652′. The realworld frustum for the car 651 is assessed from location 652 and takesinto account factors listed above such as the orientation of the car(consistent with block 503 of FIG. 5). As shown in the VIRTUAL WORLDrow, the real world frustum is applied in the virtual world to selectcandidate virtual objects (consistent with block 504). From FIG. 6E itshould be apparent that two virtual objects (depicted as cubes) fallwithin the applied frustum 653. Accordingly, these two virtual objectsmay be selected for augmentation (consistent with block 505).

At block 506, a signal is initiated to direct or control the augmentedreality output of an output device. The output device may simply be theoriginal AR device for which the viewing frustum was previouslydetermined. Depending on where the signal originates, it may betransmitted over a network such as one or more wireless networks and/orthe Internet. In this way, processing related to process 100 may beperformed on one or more remote computers (e.g., servers) of one or morecloud network, with output still being served to an end user on anetwork connected AR device. Alternatively, a single end-user device maybe configured to perform much or all of process 500, in which case thesignal initiated at block 506 may be initiated by a processor of thedevice and transmitted over a hardware connection to an output elementsuch as a display (e.g., digital screen).

At block 507, the augmented reality is ultimately output to the user.Here, the signal of block 506 is used by an output device such as a headmounted display (HMD) or a digital display to show the augmentationstogether with real world content. The augmentations may include visualaugmentations which are superimposed on the real world view.Significantly, the visual augmentations may form a visual trail or pathconfigured for a user to follower.

FIGS. 6A to 6F show two alternative augmented reality contents.Augmented reality may by definition include both real world content andvirtual world content. Real world content may be passively present, aswith see-through head mounted displays (HMDs). Alternatively, real worldcontent may be captured with a camera and subsequently displayed (e.g.,on a screen). The virtual content (i.e., augmentations) may be suppliedas overlays or otherwise imbedded with the real world video content. Toassist in distinguishing types of content illustrated in the drawings,AR1 and AR2 are illustrated such that real world content is shown inbroken lines and virtual content (i.e., augmentations) is shown in solidlines. In reality, the real world content would substantially resemblethe real world, and the augmentations would appear however embodimentsare configured to make them appear (e.g., as described in this exampleor elsewhere in this disclosure).

Note the perspective from which this content may be shown to a user mayvary. As depicted for a reader of this disclosure, both AR1 and AR2 usethe same third-person elevated perspective as used for depicting thereal world and virtual world. In general, AR outputs according toexemplary embodiments may take any of a variety of perspectives,including third-person, first-person, top-down, aerial, elevated,others, or some combination of these. FIG. 9, for example, shows ascreen capture of AR content from an elevated third-person perspective.FIGS. 10, 11, and 12 show screen captures of AR content fromfirst-person perspectives.

AR1 shows AR content which is based on changes in proximity of the truckwith respect to real world locations. Thus, in the case of AR1, thetruck is treated as a tracked target. Note that in the case of AR1,steps involving viewing frustums have not been illustrated. AR1 is abreadcrumbs-type AR in which the tracked target—the truck—appears toleave a trail of “crumbs” in the form of location-specificaugmentations. The augmentations are configured in both position andappearance based on information from their corresponding respectivevirtual objects. In this example, the closer an augmentation to thetracked target, the larger the augmentation. Conversely, the further anaugmentation from the tracked target, the smaller the augmentation. Thisis but one illustrative example.

AR2 is AR content generated on the basis of the car 651 being both atracked target and a user to whom the AR content may be provided an ARoutput. (Specifically, the user may be the car or its operator, forexample.) AR2 is an AR content which is based on changes in proximity ofthe car 651 with respect to reference locations associated with virtualobjects 611, 621, 631, and 641. Note that the virtual objects 611, 621,631, and 641 were nevertheless created on the basis of the truck 610being a tracked target. This merely illustrates that multiple targetsmay be tracked, for similar or for different reasons, to ultimatelyyield AR content for a particular embodiment.

The location of car 651 is indicated in the virtual world as a darkcircle to distinguish it from the “x” indicia for locations of the truck610. As discussed above, which of the virtual objects are candidates foruse in generating the AR content varies depending on the applied frustum653 or 653′. As was discussed above, virtual objects 611 and 621 wereselected at time T5 (FIG. 6E). Accordingly, virtual object 611 and 621are used to generate augmentations 681 and 682, respectively. At T6,shown in FIG. 6F, the same method 500 from FIG. 5 may be employed toapply the frustum 653′, select the virtual object 631, and generate acorresponding augmentation 683. As discussed elsewhere in thisdisclosure, a variety of information stored with the virtual objects maybe used in determining the particular attributes of the augmentations.In this particular example, of the candidate objects, virtual object 611has the closest proximity to the car 651 (the target). This informationis signified by making augmentation 681 blink or flash whereasaugmentation 682 does not blink or flash.

Thus far in the disclosure, attention has been focused on methods orprocesses which allow for augmented realities involving navigation,trail forming, trail following, area clearing, and the like. Suchexemplary processes are generally carried out by some combination ofhardware, software, and firmware, either in a particular electronicsdevice or by a system of electronic devices. FIG. 7 is a schematic of anexemplary system 700 for providing an augmented reality. Electronicdevices 701 may include sensors for collecting data about a user'sphysical location and pose (position and orientation). In particular,the data may reflect the location and pose of a device 701 itself. Thedevices 701 may comprise one or more cameras for capturing a real worldview of a geographic space. The captured/collected data may be sent tothe cloud 703 (e.g., processors of one or more geographically remoteservers) for data processing (e.g., frustum determination, applicationof real world frustum to virtual world, virtual object candidateidentification, augmentation selection, augmentation modification,etc.). In addition or as an alternative to a centralized cloud-basedsystem, processors involved in performing steps of processes of theinvention may be part of a decentralized system, e.g. an ad hoc systemthat varies from one embodiment or even one user to the next, at leastin terms of the particular hardware providing the computational powerfor carrying out the described methods. Databases 705 (which may be onedatabase or many) provide permanent or semi-permanent storage andretrieval for network relevant data, virtual world geometric data, othervirtual world data, virtual object data, and essentially every otherdata discussed herein which is not being newly collected from thesensors and cameras which may be deployed in the field. It should beappreciated that the various data types discussed herein which aregenerally stored in the databases, in particular virtual object data,may be updated over time when new data becomes available or existingdata becomes outdated or expired. Virtual objects and augmentationsbased on those virtual objects may be correspondingly updated. Theprocessors may use an image or video feed from the devices 701 andsensor data, in particular location data, to generate other data.Augmented image data may be sent back to the devices 701 (or otheroutput device as appropriate) which generates the augmented image 706 onits display device.

An “output device”, as used herein, may be a device capable of providingat least visual, audio, audiovisual, or tactile output to a user suchthat the user can perceive the output using his senses (e.g., using hereyes and/or ears). In many embodiments, an output device will compriseat least one display, at least one speaker, or some combination ofdisplay(s) and speaker(s). A suitable display (i.e., display device) isa screen of a mobile electronic device (e.g., phone, smartphone, GPSdevice, laptop, tablet, smartwatch, etc.). Another suitable outputdevice is a head-mounted display (HMD). In some embodiments, the displaydevice is a see-through HMD. In such cases the display device passivelypermits viewing of the real world without reproducing details of acaptured real world image feed on a screen. In a see-through HMD, it isgenerally be only the augmentations that are actively shown or output bythe device. Visual augmentations are in any case superimposed on thedirect view of the real world environment, without necessarily involvingthe display of any of the original video input to the system. In fact,for systems which do not use the video input to detect image data, thesystem may include one or more HMDs that have no camera at all, relyingentirely on other sensors (e.g. GPS, gyro, compass) to determine therelevant augmentations, and displaying them on otherwise transparentglasses or visors. Output devices and viewing devices may include or beaccompanied by input devices (e.g., buttons, touchscreens, menus,keyboards, data ports, etc.) for receiving user inputs.

FIGS. 8A and 8B show different views of an electronic device 801configured for providing an AR output 706 (FIG. 7). FIG. 8A shows anelectronic device 801 (e.g., a mobile phone) with a display 814 (e.g., ascreen) showing an output according to an example embodiment of theinvention. The electronic device 801 includes a speaker 802 as anadditional output device besides the display 814. The display 814 andthe speaker 802 are both accurately regarded as “output devices”, as isthe entire electronic device 801 by virtue of the display and speakerbeing integral therewith. The electronic device 801 comprises or isconnected with additional sensors such as an accelerometer 806,gyroscope 807, magnetic field sensor or magnetometer 808, proximitysensor 809, barometer 810, thermometer 811, and microphone 812. Thesensors collect the respective data of their respective types (e.g.,magnetometer collects magnetic field data or compass data).

An image of a real world view of a geographic space may be capturedusing one or more cameras. FIG. 8B shows a camera 804 on the rear sideof the electronic device 801. As used herein, “camera” may be a devicecapable of capturing and characterizing incident electromagneticradiation (i.e., “light”) so as to recreate a visual image as aphotograph or a series of images forming a film or video. Cameras ofsome embodiments capture only the visible spectrum (i.e., what humanssee naturally). While general consumer cameras concern only the visualspectrum detectable by the unaided human eye, other embodiments of theinvention may use one or more cameras which are capable of capturingwavelengths of light which are not visible to unaided human eyes, forinstance infrared or ultraviolet light. For the example of FIGS. 8A and8B, the geographic space that has been imaged is a portion of a streetthat includes the street itself, crosswalks, and buildings along thestreet. The image is based on incident light in the visible spectrum.The image (or images) captured by the camera is characterized by datathat describes both contents of the image (e.g., colors, pixels, etc.)and aspects of the image's capture. The capture of an image ischaracterizable with pose (which includes both position and orientation)and field of view.

A real world image or view may include (e.g., if from a city's streetintersection camera for instance) HUD displays of date and time, or evencould have augmentations in it from another augmented reality systemthat is providing video to a system based on the present disclosure.Input to one or more processors herein which is described as an image ofa real world view may also or alternatively include one or more imageswhich are not of a real world view. In general an augmented realitysystem need only have some portion of its input that is real. In someembodiments this may be a relatively small portion. Augmented realitysystems may be used to modify the augmentations of other augmentedreality systems in more complex applications, e.g., a system comprisesdistributed independent augmentation engines which make use of eachother's output.

The data from the camera(s) 804 and collected by the other sensors(e.g., 806, 807, 808, 809, 810, and/or 811) is received by one or moreprocessors 805. The camera data describes an image (or images) of a realworld view of the geographic space in the vicinity of the camera and, insome but not all embodiments, in the vicinity of a user. In thisexample, the camera 804 and the display 814 are part of the same unitaryelectronic device 801, and the geographic space is also in the vicinityof the output device, display 814. The camera 804 and the electronicdevice 800 that includes the camera 804 may be regarded as the viewingdevice. Viewing devices may include various types (but not necessarilyall types) of cameras, mobile electronic devices, mobile phones,tablets, portable computers, wearable technology, and the like. If theelectronic device 801 were a head-mounted display (HMD), the HMD wouldbe characterizable as a viewing device, too. A HMD that has no cameras,such as some see-through HMDs, may still qualify as a viewing device. Alens or pair of lenses of the see-through head-mounted display alsoqualifies as a viewing device.

A user may be able to view and benefit from what is shown by an outputdevice, e.g. display 814, in real time. The real world view captured bythe camera may be from the viewpoint of a human user as if the user weresituated in the space (e.g., sitting, standing, walking, driving,biking, etc.). In many but not all embodiments, the user is situated inthe space. A display is but one type of output device usable forproviding augmentations. Displays, speakers, and vibratory devices aredifferent examples of output devices usable in embodiments of theinvention for providing augmentation outputs to a user detectable withtheir senses. In some embodiments a viewing device and an output deviceare the same device or part of the same device. For instance, an HMD maybe accurately characterized as both a viewing device and an outputdevice, as may a mobile phone or tablet that has both a camera and adisplay screen. Alternatively, viewing devices and output devices may beseparate devices arranged at completely separate locations. A camera andsensors which are part of a viewing device collecting data about a realworld view may be a first location and an output device like a displayand/or speaker which provides augmentations with a reproduction of thereal world view may be at a second and separate location at somedistance apart from the first location.

The one or more processors 805 are configured to process the data fromthe one or more cameras 804, as well as other data like data fromsensors 806, 807, 808, 809, 810, and/or 811, in order to generate anoutput useable by an output device to present an augmented reality to auser. In some embodiments, the image and/or sensor data from thecameras/sensors is sent over a network 703 (e.g., the Internet) to oneor more remote servers comprising some of one or more processors thatperform processing of the data before augmentations are provided to anoutput device for outputting to a user.

Exemplary systems, devices, and methods according to some exemplaryembodiments provide augmented reality outputs comprising visualaugmentations which collectively describe a path used or useable totraverse a real world landscape. For illustrative purposes, FIGS. 9 to12 are frames of exemplary augmented reality outputs.

FIG. 9 is a frame of an augmented reality output in which an urban areahas been augmented with markers which serve as “breadcrumbs” from astarting location presently off the screen to the left-hand side to adestination location right of center on screen. In this example, theaugmentations appear as approximately five or six foot high verticalposts which trace a path that leads across a crosswalk to a sidewalk infront of a restaurant. Besides tracing a clear path for a user tofollow, in this case to reach the restaurant, the augmentations may alsohave various visual qualities which convey additional information. Forinstance, the sequential augmentations may be color coded to signify howclose each respective augmentation is from an end of the path.Augmentations which are furthest from the destination may be coloredred, augmentations closer to the destination may be colored orange,augmentations still closer to the destination may be colored yellow, andaugmentations nearest to the destination may be colored green, forexample.

FIGS. 10 and 11 are separate frames of separate augmented realityoutputs in which wooded or park areas have been augmented with trailposts as augmentations. These examples illustrate controlling a size ofrespective augmentations based on a distance from the AR device.Augmentations which mark trail locations which are nearest to the userare configured to be larger in size as compared with augmentations whichmark trail locations which are further away from the user.

FIG. 12 is a frame of an augmented reality output of a dividing biketrail which has been augmented with trail posts. In FIG. 12, the trailmarkers are color coded to provide directional instructions involvingalternative trails. In the viewing frustum is a division in the realworld trail, with one real world trail leading left and another realworld trail leading right. Based on a preset destination input that isentered into the system (e.g., by the user), the system may select anoptimal path among a plurality of stored paths. The alternative pathsmay be displayed with augmentations simultaneously, and an optimal pathcoded to signify its preferential treatment. In FIG. 12, for example,the path marker augmentations leading to the left at the split are codedyellow or red, while the path marker augmentations leading to the rightare color coded green. Other codings, such as with augmentation shape orsize, may be used as an alternative or as an addition to color coding.

Some embodiments of the present invention may be a system, a device, amethod, and/or a computer program product. A system, device, or computerprogram product may include a computer readable storage medium (ormedia) having computer readable program instructions thereon for causinga processor to carry out aspects of the present invention, e.g.,processes or parts of processes or a combination of processes describedherein.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Processes described herein, or steps thereof, may be embodied incomputer readable program instructions which may be paired with ordownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions and in various combinations.

These computer readable program instructions may be provided to one ormore processors of one or more general purpose computers, specialpurpose computers, or other programmable data processing apparatuses toproduce a machine or system, such that the instructions, which executevia the processor(s) of the computer or other programmable dataprocessing apparatus, create means for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks. Thesecomputer readable program instructions may also be stored in a computerreadable storage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the invention has been described herein in connection withexemplary embodiments and features, one skilled in the art willrecognize that the invention is not limited by the disclosure and thatvarious changes and modifications may be made without departing from thescope of the invention as defined by the appended claims.

1. A computer-implemented method of augmented reality (AR), comprisingassociating a plurality of virtual objects with real world locations;tracking proximity of a real world target with respect to one or more ofthe real world locations associated with the plurality of virtualobjects; and changing one or more augmentations of AR content for an ARoutput based on one or more changes in the proximity of the trackedtarget to the real world locations associated with the plurality ofvirtual objects, wherein an augmentation changed in the changing stepcorresponds with one of the plurality of virtual objects.
 2. The methodof claim 1, further comprising outputting to a user AR contentcomprising one or more augmentations which trace a path taken by thereal world target.
 3. The method of claim 1, further comprisingoutputting to a user AR content comprising one or more augmentations thepresence of which signifies that the tracked target has not visited oneor more real world locations within some predetermined time period. 4.The method of claim 1, wherein the changing step changes an appearanceof a given virtual object based on one or more of the following: whetherthe target visited the real world location associated with the givenvirtual object; time elapsed since the target visited the real worldlocation associated with the given virtual object; an amount of time thetarget spent at the real world location associated with the givenvirtual object; an identity of the target; whether a user visited thereal world location associated with the given virtual object since thetarget visited the real world location associated with the given virtualobject; whether the real world target's view included the real worldlocation associated with the given virtual object; and a real timedistance between the real world location associated with the givenvirtual object and a real time location of the target.
 5. The method ofclaim 1, wherein the target is a person, a vehicle, or a mobileelectronic device.
 6. The method of claim 1, further comprising a stepof creating the plurality of virtual objects in a virtual model.
 7. Asystem of augmented reality (AR), comprising one or more processorsconfigured to execute computer readable program instructions which, whenexecuted, cause the one or more processors to perform associating aplurality of virtual objects with real world locations; trackingproximity of a real world target with respect to one or more of the realworld locations associated with the plurality of virtual objects; andchanging one or more augmentations of AR content for an AR output basedon one or more changes in the proximity of the tracked target to thereal world locations associated with the plurality of virtual objects,wherein an augmentation changed in the changing step corresponds withone of the plurality of virtual objects; and at least one output devicefor outputting the AR content to a user.
 8. The system of claim 7,wherein the one or more processors are caused by the computer readableprogram instructions to make one or more augmentations of the AR outputtrace a path taken by the real world target.
 9. The system of claim 7,wherein the one or more processors are caused by the computer readableprogram instructions to make a presence of one or more augmentations inthe AR output signify that the tracked target has not visited one ormore real world locations within some predetermined time period.
 10. Thesystem of claim 7, wherein the changing step changes an appearance of agiven virtual object based on one or more of the following: whether thetarget visited the real world location associated with the given virtualobject; time elapsed since the target visited the real world locationassociated with the given virtual object; an amount of time the targetspent at the real world location associated with the given virtualobject; an identity of the target; whether a user visited the real worldlocation associated with the given virtual object since the targetvisited the real world location associated with the given virtualobject; whether the real world target's view included the real worldlocation associated with the given virtual object; and a real timedistance between the real world location associated with the givenvirtual object and a real time location of the target.
 11. The system ofclaim 7, wherein the target is a person, a vehicle, or a mobileelectronic device.
 12. The system of claim 7, further comprising avirtual model for containing the plurality of virtual objects.
 13. Acomputer readable program product comprising a non-transitory computerreadable medium with computer readable instructions which, when executedby one or more processors, cause the one or more processors to performassociating a plurality of virtual objects with real world locations;tracking proximity of a real world target with respect to one or more ofthe real world locations associated with the plurality of virtualobjects; and changing one or more augmentations of AR content for an ARoutput based on one or more changes in the proximity of the trackedtarget to the real world locations associated with the plurality ofvirtual objects, wherein an augmentation changed in the changing stepcorresponds with one of the plurality of virtual objects.
 14. Thecomputer readable program product of claim 13, further comprisingcomputer readable instructions which cause the one or more processors toperform outputting to a user AR content comprising one or moreaugmentations which trace a path taken by the real world target.
 15. Thecomputer readable program product of claim 13, further comprisingcomputer readable instructions which cause the one or more processors toperform outputting to a user AR content comprising one or moreaugmentations, wherein the presence of augmentations signifies that thetracked target has not visited one or more real world locations withinsome predetermined time period.
 16. The computer readable programproduct of claim 13, wherein the changing step changes an appearance ofa given virtual object based on one or more of the following: whetherthe target visited the real world location associated with the givenvirtual object; time elapsed since the target visited the real worldlocation associated with the given virtual object; an amount of time thetarget spent at the real world location associated with the givenvirtual object; an identity of the target; whether a user visited thereal world location associated with the given virtual object since thetarget visited the real world location associated with the given virtualobject; whether the real world target's view included the real worldlocation associated with the given virtual object; and a real timedistance between the real world location associated with the givenvirtual object and a real time location of the target.
 17. The computerreadable program product of claim 13, wherein the target is a person, avehicle, or a mobile electronic device.
 18. The computer readableprogram product of claim 13, further comprising computer readableinstructions which cause the one or more processors to perform creatingthe plurality of virtual objects in a virtual model. 19-30. (canceled)31. The method of claim 1, further comprising providing a 3D virtualmodel having virtual locations configured to correspond with the realworld locations, wherein the plurality of virtual objects are in the 3Dvirtual model; as the real world target moves from a first real worldlocation to a second real world location, adding or updating one or morevirtual objects within the 3D virtual model at virtual world locationsmatching the first and second real world locations.
 32. The system ofclaim 7, wherein the one or more processors are caused by the computerreadable program instructions to provide a 3D virtual model havingvirtual locations configured to correspond with the real worldlocations, wherein the plurality of virtual objects are in the 3Dvirtual model; as the real world target moves from a first real worldlocation to a second real world location, add or update one or morevirtual objects within the 3D virtual model at virtual world locationsmatching the first and second real world locations.
 33. The computerreadable program product of claim 13, further comprising computerreadable instructions which cause the one or more processors to performproviding a 3D virtual model having virtual locations configured tocorrespond with the real world locations, wherein the plurality ofvirtual objects are in the 3D virtual model; as the real world targetmoves from a first real world location to a second real world location,adding or updating one or more virtual objects within the 3D virtualmodel at virtual world locations matching the first and second realworld locations.